{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GEKKO Python Example Applications\n",
    "\n",
    "![GEKKO Optimization Suite](http://apmonitor.com/wiki/uploads/Main/gekko_optimization.png)\n",
    "\n",
    "GEKKO is optimization software for mixed-integer and differential algebraic equations. It is coupled with large-scale solvers for linear, quadratic, nonlinear, and mixed integer programming (LP, QP, NLP, MILP, MINLP). Modes of operation include data reconciliation, real-time optimization, dynamic simulation, and nonlinear predictive control. See the [GEKKO documentation](http://gekko.readthedocs.io/en/latest/overview.html) for additional information.\n",
    "\n",
    "1. Solver Selection\n",
    "2. Solve Linear Equations\n",
    "3. Solve Nonlinear Equations\n",
    "4. Interpolation with Cubic Spline\n",
    "5. Linear and Polynomial Regression\n",
    "6. Nonlinear Regression\n",
    "7. Machine Learning / Artificial Neural Network\n",
    "8. Solve Differential Equation(s)\n",
    "9. Nonlinear Programming Optimization\n",
    "10. Mixed Integer Nonlinear Programming\n",
    "11. Optimal Control with Integral Objective\n",
    "12. Optimal Control with Economic Objective\n",
    "13. Optimal Control: Minimize Final Time\n",
    "14. PID Control Tuning\n",
    "15. Process Simulator\n",
    "16. Moving Horizon Estimation\n",
    "17. Model Predictive Control\n",
    "18. Debugging Resources\n",
    "\n",
    "This is a blank workbook that is filled in with [solutions available on GitHub](https://github.com/BYU-PRISM/GEKKO/blob/master/docs/gekko_examples.ipynb) or through an online [notebook viewer](https://nbviewer.jupyter.org/github/BYU-PRISM/GEKKO/blob/master/docs/gekko_examples.ipynb). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install and import GEKKO\n",
    "\n",
    "# other packages needed in this notebook\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1: Solver selection\n",
    "\n",
    "Solve $y^2=1$ with APOPT solver. See [APMonitor documentation](https://apmonitor.com/wiki/index.php/Main/OptionApmSolver) or [GEKKO documentation](http://gekko.readthedocs.io/en/latest/global.html?highlight=solver#solver) for additional solver options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2: Solve Linear Equations\n",
    "\n",
    "$3\\,x + 2\\,y=1$\n",
    "\n",
    "$x+2\\,y=0$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3: Solve Nonlinear Equations\n",
    "\n",
    "$x+2\\,y=0$\n",
    "\n",
    "$x^2+y^2=1$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4: Interpolation with Cubic Spline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAADntJREFUeJzt3X+I5Pddx/Hna5OGuv1h/7hVwt3t\nbv64iqGIKUsoBLRqK5dYLv4hkrBRlNL9p9FKi5KyUjVy/1RQEeKPQUupjg3R+uOo0Vg0pVaa9vba\nNPZyRo7Y3SxXuOtPLYvG6Ns/Zq/Zu+7dzNzN7Nx89vmA47vfz3wz8x7CPe/Ld2Z2UlVIktoyM+kB\nJEmjZ9wlqUHGXZIaZNwlqUHGXZIaZNwlqUHGXZIaZNwlqUHGXZIadPOkHvjAgQO1uLg4qYeXpKl0\n6tSpL1fVXL/jJhb3xcVF1tbWJvXwkjSVkqwPcpyXZSSpQcZdkhpk3CWpQcZdkhpk3CWpQX3jnuQD\nSc4n+cIVbk+S30lyNskzSd44+jEltabbhcVFmJnpbbvdSU/UlkHO3D8IHL3K7XcDR7b/rAC/d/1j\nSWpZtwsrK7C+DlW97cqKgR+lvnGvqk8AX73KIfcCH6qep4DXJbl1VANKas/qKmxtXbq2tdVb12iM\n4pr7QeCFHfub22vfJslKkrUkaxcuXBjBQ0uaRhsbw61reKOIe3ZZ2/Vbt6uqU1VLVbU0N9f307OS\nGjU/P9y6hjeKuG8Ch3fsHwLOjeB+JTXq+HGYnb10bXa2t67RGEXcTwA/vf2umTcB36iqL43gfiU1\nankZOh1YWICkt+10eusajb6/OCzJh4E3AweSbAK/ArwCoKp+H3gcuAc4C2wBPzuuYSW1Y3nZmI9T\n37hX1f19bi/gnSObSJJ03fyEqiQ1yLhLUoOMuyQ1yLhLUoOMuyQ1yLhLUoOMuyQ1yLhLUoOMuyQ1\nyLhLUoOMuyQ1yLhLUoOMuyQ1yLhL0h7pdmFxEWZmettxfiF431/5K0m6ft0urKy8/MXg6+u9fRjP\n77X3zF2S9sDq6sthv2hrq7c+DsZdkvbAxsZw69fLuEvSHpifH279ehl3SdoDx4/D7Oyla7OzvfVx\nMO6StAeWl6HTgYUFSHrbTmd8XxLuu2UkaY8sL48v5pfzzF2SGmTcJalBxl2SGmTcJalBxl2SGmTc\nJalBxl2SGmTcJalBxl2SGmTcJalBxl2SGmTcJalBxl2SGmTcJalBA8U9ydEkzyU5m+ShXW6fT/Jk\nks8leSbJPaMfVZI0qL5xT3IT8AhwN3A7cH+S2y877JeBx6rqDuA+4HdHPagkaXCDnLnfCZytquer\n6kXgUeDey44p4LXbP38ncG50I0qShjVI3A8CL+zY39xe2+lXgQeSbAKPAz+32x0lWUmylmTtwoUL\n1zCuJGkQg8Q9u6zVZfv3Ax+sqkPAPcAfJ/m2+66qTlUtVdXS3Nzc8NNKkgYySNw3gcM79g/x7Zdd\n3g48BlBVnwJeCRwYxYCSpOENEveTwJEktyW5hd4LpicuO2YD+BGAJN9LL+5ed5GkCekb96p6CXgQ\neAI4Q+9dMaeTPJzk2PZh7wHekeTzwIeBn6mqyy/dSJL2yM2DHFRVj9N7oXTn2vt2/PwscNdoR5Mk\nXSs/oSpJDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg\n4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5J\nDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktSggeKe5GiS55KcTfLQFY75\nySTPJjmd5E9HO6YkaRg39zsgyU3AI8BbgU3gZJITVfXsjmOOAO8F7qqqryX5rnENLEnqb5Az9zuB\ns1X1fFW9CDwK3HvZMe8AHqmqrwFU1fnRjilJGsYgcT8IvLBjf3N7bafXA69P8s9JnkpydFQDSpKG\n1/eyDJBd1mqX+zkCvBk4BPxTkjdU1dcvuaNkBVgBmJ+fH3pYSdJgBjlz3wQO79g/BJzb5Zi/rqr/\nqap/B56jF/tLVFWnqpaqamlubu5aZ5Yk9TFI3E8CR5LcluQW4D7gxGXH/BXwQwBJDtC7TPP8KAeV\nJA2ub9yr6iXgQeAJ4AzwWFWdTvJwkmPbhz0BfCXJs8CTwC9W1VfGNbQk6epSdfnl872xtLRUa2tr\nE3lsSZpWSU5V1VK/4/yEqiQ1yLhLUoOMuyQ1yLhLUoOMuyQ1yLhLUoOMuyQ1yLhLUoOMuyQ1yLhL\nUoOMuyQ1yLhLUoOMuyQ1yLhLUoOMu3QD6HZhcRFmZnrbbnfSE2naDfIdqpLGqNuFlRXY2urtr6/3\n9gGWlyc3l6abZ+7ShK2uvhz2i7a2euvStTLu0oRtbAy3Lg3CuEsTNj8/3Lo0COMuTdjx4zA7e+na\n7GxvXbpWxl2asOVl6HRgYQGS3rbT8cVUXR/fLSPdAJaXjblGyzN3SWqQcZekBhl3SWqQcZekBhl3\nSWqQcZekBhl3SWqQcZekBhl3SWqQcZekBhl3SWqQcZekBhl3SWrQQHFPcjTJc0nOJnnoKsf9RJJK\nsjS6ESVJw+ob9yQ3AY8AdwO3A/cnuX2X414D/Dzw6VEPKUkaziBn7ncCZ6vq+ap6EXgUuHeX434d\neD/wXyOcT5J0DQaJ+0HghR37m9tr35LkDuBwVX10hLNJkq7RIHHPLmv1rRuTGeC3gPf0vaNkJcla\nkrULFy4MPqUkaSiDxH0TOLxj/xBwbsf+a4A3AB9P8kXgTcCJ3V5UrapOVS1V1dLc3Ny1Ty1JuqpB\n4n4SOJLktiS3APcBJy7eWFXfqKoDVbVYVYvAU8Cxqloby8SSpL76xr2qXgIeBJ4AzgCPVdXpJA8n\nOTbuASVJw7t5kIOq6nHg8cvW3neFY998/WNJkq6Hn1CVpAYZd0lqkHGXpAYZd0lqkHGXpAYZd0lq\nkHGXpAYZd0lqkHGXpAYZd0lqkHGXpAYZd0lqkHGXpAYZd0lqkHHXDafbhcVFmJnpbbvdSU8kTZ+B\nfp+7tFe6XVhZga2t3v76em8fYHl5cnNJ08Yzd91QVldfDvtFW1u9dUmDM+66oWxsDLcuaXfGXTeU\n+fnh1iXtzrjrhnL8OMzOXro2O9tblzQ4464byvIydDqwsABJb9vp+GKqNCzfLaMbzvKyMZeul2fu\nktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktSg\ngeKe5GiS55KcTfLQLre/O8mzSZ5J8g9JFkY/qiRpUH3jnuQm4BHgbuB24P4kt1922OeApar6PuDP\ngfePetD9qtuFxUWYmeltu91JTyRpGgxy5n4ncLaqnq+qF4FHgXt3HlBVT1bVxa81fgo4NNox96du\nF1ZWYH0dqnrblRUDL6m/QeJ+EHhhx/7m9tqVvB342+sZSj2rq7C1dena1lZvXZKuZpBvYsoua7Xr\ngckDwBLwg1e4fQVYAZj3G4/72tgYbl2SLhrkzH0TOLxj/xBw7vKDkrwFWAWOVdV/73ZHVdWpqqWq\nWpqbm7uWefeVK/3757+LkvoZJO4ngSNJbktyC3AfcGLnAUnuAP6AXtjPj37M/en4cZidvXRtdra3\nLklX0zfuVfUS8CDwBHAGeKyqTid5OMmx7cN+A3g18GdJnk5y4gp3pyEsL0OnAwsLkPS2nY5fHi2p\nv1Ttevl87JaWlmptbW0ijy1J0yrJqapa6necn1CVpAYZd0lqkHGXpAYZd0lqkHGXpAYZd0lqkHGX\npAYZd0lqkHGXpAYZd0lqkHGXpAYZd0lqkHGXpAYZd0lq0FTFvduFxUWYmelt/aJoSdrdIN+hekPo\ndmFl5eUvjF5f7+2DX14hSZebmjP31dWXw37R1lZvXZJ0qamJ+8bGcOuStJ9NTdzn54dbl6T9bGri\nfvw4zM5eujY721uXJF1qauK+vAydDiwsQNLbdjq+mCpJu5mad8tAL+TGXJL6m5ozd0nS4Iy7JDXI\nuEtSg4y7JDXIuEtSg1JVk3ng5AKwfo3/+QHgyyMcZxr4nPcHn/P+cD3PeaGq5vodNLG4X48ka1W1\nNOk59pLPeX/wOe8Pe/GcvSwjSQ0y7pLUoGmNe2fSA0yAz3l/8DnvD2N/zlN5zV2SdHXTeuYuSbqK\nqYt7kqNJnktyNslDk55n3JJ8IMn5JF+Y9Cx7JcnhJE8mOZPkdJJ3TXqmcUvyyiSfSfL57ef8a5Oe\naS8kuSnJ55J8dNKz7IUkX0zyL0meTrI21seapssySW4C/g14K7AJnATur6pnJzrYGCX5AeCbwIeq\n6g2TnmcvJLkVuLWqPpvkNcAp4Mcb//8c4FVV9c0krwA+Cbyrqp6a8GhjleTdwBLw2qp626TnGbck\nXwSWqmrs7+uftjP3O4GzVfV8Vb0IPArcO+GZxqqqPgF8ddJz7KWq+lJVfXb75/8EzgAHJzvVeFXP\nN7d3X7H9Z3rOvK5BkkPAjwF/OOlZWjRtcT8IvLBjf5PG/9Lvd0kWgTuAT092kvHbvkTxNHAe+FhV\ntf6cfxv4JeD/Jj3IHirg75OcSrIyzgeatrhnl7Wmz272sySvBj4C/EJV/cek5xm3qvrfqvp+4BBw\nZ5JmL8MleRtwvqpOTXqWPXZXVb0RuBt45/Zl17GYtrhvAod37B8Czk1oFo3R9nXnjwDdqvqLSc+z\nl6rq68DHgaMTHmWc7gKObV+DfhT44SR/MtmRxq+qzm1vzwN/Se9S81hMW9xPAkeS3JbkFuA+4MSE\nZ9KIbb+4+EfAmar6zUnPsxeSzCV53fbP3wG8BfjXyU41PlX13qo6VFWL9P4e/2NVPTDhscYqyau2\n3yBAklcBPwqM7V1wUxX3qnoJeBB4gt6LbI9V1enJTjVeST4MfAr4niSbSd4+6Zn2wF3AT9E7m3t6\n+889kx5qzG4FnkzyDL2TmI9V1b54e+A+8t3AJ5N8HvgM8DdV9XfjerCpeiukJGkwU3XmLkkajHGX\npAYZd0lqkHGXpAYZd0lqkHGXpAYZd0lqkHGXpAb9P452dLmCy2k6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1acf9bd39b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "xm = np.array([0,1,2,3,4,5])\n",
    "ym = np.array([0.1,0.2,0.3,0.5,1.0,0.9])\n",
    "\n",
    "# create plot\n",
    "plt.plot(xm,ym,'bo')\n",
    "plt.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5: Linear and Polynomial Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'y')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFrlJREFUeJzt3X+QXWd93/H3R7KIWOHWPyQ7Hsva\ndTOejCG1hdkaWpcAGTAyUEyb/mF3ISYDs0NADWk67TjdDh6bbpshMziTQgPbRMMPr+2m/GjV1mDU\nYmqI46AVkQ22g1CFVt4RjDcW4dfaAdnf/rFX5lrelY7kPbrau+/XzJm95znPufs94/F+dJ7znHNS\nVUiSdDyrel2AJGl5MDAkSY0YGJKkRgwMSVIjBoYkqREDQ5LUiIEhSWrEwJAkNWJgSJIaOaPXBSyl\n9evX19DQUK/LkKRlY9euXX9VVRua9O2rwBgaGmJqaqrXZUjSspFkumlfh6QkSY0YGJKkRgwMSVIj\nfXUNYyE//elPmZmZ4cknn+x1Ka1au3YtGzduZM2aNb0uRVKf6vvAmJmZ4cwzz2RoaIgkvS6nFVXF\n448/zszMDBdffHGvy5HUp1obkkpyUZJ7kjyS5KEk712gT5L8QZK9SR5MckXXthuSfKuz3HCydTz5\n5JOce+65fRsWAEk499xz+/4sStKzTU5OMjQ0xKpVqxgaGmJycrLV39fmGcZh4F9W1deSnAnsSrKj\nqh7u6nMNcElneTnwh8DLk5wD3AQMA9XZd3tVfe9kCunnsDhiJRyjpJ+ZnJxkdHSUubk5AKanpxkd\nHQVgZGSkld/Z2hlGVX2nqr7W+fxD4BHgwqO6XQt8oubdD5yV5ALg9cCOqjrUCYkdwJa2apWk5WZs\nbOyZsDhibm6OsbGx1n7nKZkllWQIeCnw50dtuhB4tGt9ptO2WPtC3z2aZCrJ1Ozs7FKVvKRWr17N\n5s2beclLXsLll1/OBz/4QZ5++ulj7rN//35uv/32U1ShpOXmwIEDJ9S+FFoPjCQvAj4N/FZV/eDo\nzQvsUsdof25j1URVDVfV8IYNje5uP6Y2xgRf+MIXsnv3bh566CF27NjBXXfdxc0333zMfQwMScey\nadOmE2pfCq0GRpI1zIfFZFV9ZoEuM8BFXesbgYPHaG/VkTHB6elpquqZMcGlvJB03nnnMTExwYc+\n9CGqiv379/PKV76SK664giuuuIL77rsPgBtvvJEvf/nLbN68mVtvvXXRfpJWpvHxcQYGBp7VNjAw\nwPj4eHu/tKpaWZg/S/gE8PvH6PNG4HOdvq8AvtppPwf4NnB2Z/k2cM7xfufLXvayOtrDDz/8nLbF\nDA4OFvNnMs9aBgcHG3/HQtatW/ectrPOOqu++93v1o9//ON64oknqqpqz549deQY7rnnnnrjG9/4\nTP/F+nU7kWOVtPzddtttNTg4WElqcHCwbrvtthP+DmCqGv5db3OW1FXA24CvJ9ndafs3wKZOUH0E\nuAt4A7AXmAN+vbPtUJL3Azs7+91SVYdarBU4tWOC8/+d5m8s3Lp1K7t372b16tXs2bNnwf5N+0la\nOUZGRlqbEbWQ1gKjqr7CwtciuvsU8J5Ftm0DtrVQ2qI2bdrE9PRzH9y41GOC+/btY/Xq1Zx33nnc\nfPPNnH/++TzwwAM8/fTTrF27dsF9br311kb9JKktPkuqy6kYE5ydneVd73oXW7duJQnf//73ueCC\nC1i1ahWf/OQneeqppwA488wz+eEPf/jMfov1k6RTxcDoMjIywsTEBIODgyRhcHCQiYmJ533K98QT\nTzwzrfa1r30tV199NTfddBMA7373u/n4xz/OK17xCvbs2cO6desAuOyyyzjjjDO4/PLLufXWWxft\nJ0mnSo6MpfeD4eHhOvoFSo888giXXnppjyo6tVbSsUpaGkl2VdVwk76eYUiSGjEwJEmNrIjA6Kdh\nt8WshGOU1Ft9Hxhr167l8ccf7+s/qNV5H4ZTbSW1qe9foLRx40ZmZmY4XR9MuFSOvHFPktrS94Gx\nZs0a30InSUug74ekJElLw8CQJDViYEiSGjEwJEmNGBiSpEYMDElSIwaGJKkRA0OS1EhrN+4l2Qa8\nCXisqn5pge3/CjjyookzgEuBDZ3Xs+4Hfgg8BRxu+uhdSVJ72jzD+BiwZbGNVfV7VbW5qjYDvwP8\n36Pe2/2aznbDQpJOA60FRlXdCxw6bsd51wN3tFWLJOn56/k1jCQDzJ+JfLqruYAvJNmVZPQ4+48m\nmUoy1e8PGJSkXup5YAD/CPjTo4ajrqqqK4BrgPck+eXFdq6qiaoarqrhDRs2tF2rJK1Yp0NgXMdR\nw1FVdbDz8zHgs8CVPahLktSlp4GR5G8DrwL+e1fbuiRnHvkMXA18ozcVSpKOaHNa7R3Aq4H1SWaA\nm4A1AFX1kU63fwx8oap+3LXr+cBnkxyp7/aq+nxbdUqSmmktMKrq+gZ9Psb89Nvutn3A5e1UJUk6\nWafDNQxJ0jJgYEiSGjEwJEmNGBiSpEYMDElSIwaGJKkRA0OS1IiBIUlqxMCQJDViYEiSGjEwJEmN\nGBiSpEYMDElSIwaGJKkRA0OS1IiBIUlqxMCQJDXSWmAk2ZbksSQLvo87yauTfD/J7s7yvq5tW5J8\nM8neJDe2VaMkqbk2zzA+Bmw5Tp8vV9XmznILQJLVwIeBa4AXA9cneXGLdUqSGmgtMKrqXuDQSex6\nJbC3qvZV1U+AO4Frl7Q4SdIJ6/U1jL+f5IEkn0vykk7bhcCjXX1mOm0LSjKaZCrJ1OzsbJu1StKK\n1svA+BowWFWXA/8R+G+d9izQtxb7kqqaqKrhqhresGFDC2VKkqCHgVFVP6iqH3U+3wWsSbKe+TOK\ni7q6bgQO9qBESVKXngVGkp9Pks7nKzu1PA7sBC5JcnGSFwDXAdt7Vackad4ZbX1xkjuAVwPrk8wA\nNwFrAKrqI8A/BX4jyWHgCeC6qirgcJKtwN3AamBbVT3UVp2SpGYy/ze6PwwPD9fU1FSvy5CkZSPJ\nrqoabtK317OkJEnLhIEhSWrEwJAkNWJgSJIaMTAkSY0YGJKkRgwMSVIjBoYkqREDQ5LUiIEhSWrE\nwJAkNWJgSJIaMTAkSY0YGJKkRgwMSVIjBoYkqREDQ5LUSGuBkWRbkseSfGOR7SNJHuws9yW5vGvb\n/iRfT7I7ia/Qk6TTQJtnGB8Dthxj+7eBV1XVZcD7gYmjtr+mqjY3fXWgJKldZ7T1xVV1b5KhY2y/\nr2v1fmBjW7VIkp6/0+UaxjuAz3WtF/CFJLuSjB5rxySjSaaSTM3OzrZapCStZK2dYTSV5DXMB8Y/\n7Gq+qqoOJjkP2JHkL6vq3oX2r6oJOsNZw8PD1XrBkrRC9fQMI8llwB8B11bV40faq+pg5+djwGeB\nK3tToSTpiJ4FRpJNwGeAt1XVnq72dUnOPPIZuBpYcKaVJOnUaW1IKskdwKuB9UlmgJuANQBV9RHg\nfcC5wH9KAnC4MyPqfOCznbYzgNur6vNt1SlJaqbNWVLXH2f7O4F3LtC+D7j8uXtIknrpdJklJUk6\nzRkYkqRGDAxJUiMGhiSpEQNDktSIgSFJasTAkCQ1YmBIkhoxMCRJjRgYkqRGDAxJUiMGhiSpEQND\nktTIcQMjydYkZ5+KYiRJp68mZxg/D+xM8idJtqTzogpJ0spy3MCoqn8LXAL8MfB24FtJ/n2SX2i5\nNknSaaTRNYyqKuC7neUwcDbwqSQfaLE2SdJppMk1jN9Msgv4APCnwN+tqt8AXgb86nH23ZbksSQL\nvpM78/4gyd4kDya5omvbDUm+1VluOKGjkiQtuSavaF0P/JOqmu5urKqnk7zpOPt+DPgQ8IlFtl/D\n/HDXJcDLgT8EXp7kHObfAT4MFLAryfaq+l6DeiVJLWhyDeN9R4dF17ZHjrPvvcChY3S5FvhEzbsf\nOCvJBcDrgR1VdagTEjuALcerVZLUnl7fh3Eh8GjX+kynbbH250gymmQqydTs7GxrhUrSStfrwFho\nim4do/25jVUTVTVcVcMbNmxY0uIkST/T68CYAS7qWt8IHDxGuySpR3odGNuBX+vMlnoF8P2q+g5w\nN3B1krM7d5lf3WmTJPVIk1lSJy3JHcCrgfVJZpif+bQGoKo+AtwFvAHYC8wBv97ZdijJ+4Gdna+6\npaqOdfFcktSyVgOjqq4/zvYC3rPItm3AtjbqkiSduF4PSUmSlgkDQ5LUiIEhSWrEwJAkNWJgSJIa\nMTAkSY0YGJKkRgwMSVIjBoYkqREDQ5LUiIEhSWrEwJAkNWJgSOoLk5OTDA0NsWrVKoaGhpicnOx1\nSX2n1afVStKpMDk5yejoKHNzcwBMT08zOjoKwMjISC9L6yueYUha9sbGxp4JiyPm5uYYGxvrUUX9\nycCQtOwdOHDghNp1cgwMScvepk2bTqhdJ6fVwEiyJck3k+xNcuMC229Nsruz7Eny113bnuratr3N\nOiUtb+Pj4wwMDDyrbWBggPHx8R5V1J9au+idZDXwYeB1wAywM8n2qnr4SJ+q+hdd/f858NKur3ii\nqja3VZ+k/nHkwvbY2BgHDhxg06ZNjI+Pe8F7ibU5S+pKYG9V7QNIcidwLfDwIv2vB25qsR5JfWxk\nZMSAaFmbQ1IXAo92rc902p4jySBwMfDFrua1SaaS3J/kLYv9kiSjnX5Ts7OzS1G3JGkBbQZGFmir\nRfpeB3yqqp7qattUVcPAPwN+P8kvLLRjVU1U1XBVDW/YsOH5VSxJWlSbgTEDXNS1vhE4uEjf64A7\nuhuq6mDn5z7gSzz7+oYk6RRrMzB2ApckuTjJC5gPhefMdkryi8DZwJ91tZ2d5Oc6n9cDV7H4tQ9J\n0inQ2kXvqjqcZCtwN7Aa2FZVDyW5BZiqqiPhcT1wZ1V1D1ddCnw0ydPMh9rvds+ukiSdenn23+nl\nbXh4uKampnpdhiQtG0l2da4XH5d3ekuSGjEwJEmNGBiSpEYMDElSIwaGJKkRA0OS1IiBIUlqxMCQ\nJDViYEiSGjEwJEmNGBiSpEYMDElSIwaGJKkRA0OS1IiBIUlqxMCQJDViYEiSGmk1MJJsSfLNJHuT\n3LjA9rcnmU2yu7O8s2vbDUm+1VluaLNOqd9MTk4yNDTEqlWrGBoaYnJystclqQ+09k7vJKuBDwOv\nA2aAnUm2L/Bu7v9SVVuP2vcc4CZgGChgV2ff77VVr9QvJicnGR0dZW5uDoDp6WlGR0cBGBkZ6WVp\nWubaPMO4EthbVfuq6ifAncC1Dfd9PbCjqg51QmIHsKWlOqW+MjY29kxYHDE3N8fY2FiPKlK/aDMw\nLgQe7Vqf6bQd7VeTPJjkU0kuOsF9STKaZCrJ1Ozs7FLULS1rBw4cOKF2qak2AyMLtNVR6/8DGKqq\ny4D/DXz8BPadb6yaqKrhqhresGHDSRcr9YtNmzadULvUVJuBMQNc1LW+ETjY3aGqHq+qv+ms/mfg\nZU33lbSw8fFxBgYGntU2MDDA+Ph4jypSv2gzMHYClyS5OMkLgOuA7d0dklzQtfpm4JHO57uBq5Oc\nneRs4OpOm6TjGBkZYWJigsHBQZIwODjIxMSEF7z1vLU2S6qqDifZyvwf+tXAtqp6KMktwFRVbQd+\nM8mbgcPAIeDtnX0PJXk/86EDcEtVHWqrVqnfjIyMGBBacqla8NLAsjQ8PFxTU1O9LkOSlo0ku6pq\nuElf7/SWJDViYEiSGjEwJEmNGBiSpEYMDElSIwaGJKkRA0OS1IiBIUlqxMCQJDViYEiSGjEwJEmN\nGBiSpEYMDK0Ik5OTDA0NsWrVKoaGhpicnOx1SdKy09rjzaXTxeTkJKOjo8+853p6eprR0VEAHwEu\nnQDPMNT3xsbGngmLI+bm5hgbG+tRRdLyZGCo7x04cOCE2iUtzMBQ39u0adMJtUtaWKuBkWRLkm8m\n2ZvkxgW2/3aSh5M8mOT/JBns2vZUkt2dZfvR+0pNjY+PMzAw8Ky2gYEBxsfHe1SRtDy1FhhJVgMf\nBq4BXgxcn+TFR3X7C2C4qi4DPgV8oGvbE1W1ubO8ua061f9GRkaYmJhgcHCQJAwODjIxMeEFb+kE\ntTlL6kpgb1XtA0hyJ3At8PCRDlV1T1f/+4G3tliPVrCRkREDQnqe2hySuhB4tGt9ptO2mHcAn+ta\nX5tkKsn9Sd6y2E5JRjv9pmZnZ59fxZKkRbV5hpEF2mrBjslbgWHgVV3Nm6rqYJK/A3wxyder6v89\n5wurJoAJgOHh4QW/X5L0/LV5hjEDXNS1vhE4eHSnJK8FxoA3V9XfHGmvqoOdn/uALwEvbbFWSdJx\ntBkYO4FLklyc5AXAdcCzZjsleSnwUebD4rGu9rOT/Fzn83rgKrqufej58TEZkk5Ga0NSVXU4yVbg\nbmA1sK2qHkpyCzBVVduB3wNeBPzXJAAHOjOiLgU+muRp5kPtd6vKwFgCPiZD0slKVf8M+w8PD9fU\n1FSvyzitDQ0NMT09/Zz2wcFB9u/ff+oLktRTSXZV1XCTvt7pvcL4mAxJJ8vAWGF8TIakk2VgrDA+\nJkPSyTIwVhgfkyHpZHnRW5JWMC96S5KWnIEhSWpkxQeGdz1LUjNtPnzwtOddz5LU3Io+wxgbG3sm\nLI6Ym5tjbGysRxVJ0ulrRQeGdz1LUnMrOjC861mSmlvRgeFdz5LU3IoODO96lqTmvNNbklYw7/SW\nJC05A0OS1IiBIUlqxMCQJDViYEiSGumrWVJJZoHpk9x9PfBXS1jOcuAx97+VdrzgMZ+owara0KRj\nXwXG85FkqunUsn7hMfe/lXa84DG3ySEpSVIjBoYkqRED42cmel1AD3jM/W+lHS94zK3xGoYkqRHP\nMCRJjaz4wEiyJck3k+xNcmOv6zkVkmxL8liSb/S6llMhyUVJ7knySJKHkry31zW1LcnaJF9N8kDn\nmG/udU2nSpLVSf4iyf/sdS2nQpL9Sb6eZHeSVp++uqKHpJKsBvYArwNmgJ3A9VX1cE8La1mSXwZ+\nBHyiqn6p1/W0LckFwAVV9bUkZwK7gLf083/nJAHWVdWPkqwBvgK8t6ru73FprUvy28Aw8Leq6k29\nrqdtSfYDw1XV+r0nK/0M40pgb1Xtq6qfAHcC1/a4ptZV1b3AoV7XcapU1Xeq6mudzz8EHgEu7G1V\n7ap5P+qsruksff+vwyQbgTcCf9TrWvrRSg+MC4FHu9Zn6PM/JCtdkiHgpcCf97aS9nWGZnYDjwE7\nqqrvjxn4feBfA0/3upBTqIAvJNmVZLTNX7TSAyMLtPX9v8JWqiQvAj4N/FZV/aDX9bStqp6qqs3A\nRuDKJH09/JjkTcBjVbWr17WcYldV1RXANcB7OkPOrVjpgTEDXNS1vhE42KNa1KLOOP6ngcmq+kyv\n6zmVquqvgS8BW3pcStuuAt7cGdO/E/iVJLf1tqT2VdXBzs/HgM8yP9TeipUeGDuBS5JcnOQFwHXA\n9h7XpCXWuQD8x8AjVfXBXtdzKiTZkOSszucXAq8F/rK3VbWrqn6nqjZW1RDz/y9/sare2uOyWpVk\nXWciB0nWAVcDrc1+XNGBUVWHga3A3cxfCP2Tqnqot1W1L8kdwJ8Bv5hkJsk7el1Ty64C3sb8vzh3\nd5Y39Lqoll0A3JPkQeb/YbSjqlbENNMV5nzgK0keAL4K/K+q+nxbv2xFT6uVJDW3os8wJEnNGRiS\npEYMDElSIwaGJKkRA0OS1IiBIUlqxMCQJDViYEgtSfL3kjzYeTfFus57Kfr6eU7qb964J7Uoyb8D\n1gIvBGaq6j/0uCTppBkYUos6zyjbCTwJ/IOqeqrHJUknzSEpqV3nAC8CzmT+TENatjzDkFqUZDvz\nj9q+mPnXxG7tcUnSSTuj1wVI/SrJrwGHq+r2zvvj70vyK1X1xV7XJp0MzzAkSY14DUOS1IiBIUlq\nxMCQJDViYEiSGjEwJEmNGBiSpEYMDElSIwaGJKmR/w9DOZzcz0uDHwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1acf9bd37b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "xm = np.array([0,1,2,3,4,5])\n",
    "ym = np.array([0.1,0.2,0.3,0.5,0.8,2.0])\n",
    "\n",
    "# plot data\n",
    "plt.plot(xm,ym,'ko',label='Data')\n",
    "plt.legend(loc='best')\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('y')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6: Nonlinear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1acfa0278d0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFMNJREFUeJzt3X+MXeWd3/H3B2PWsUPLDw8UYeyh\nFVqRrMCQK0JFs0kqQkxIIVX7B9RJ2VUiK03cZrtVK7ZIiyBCWnWlEG2TbjK7tUiCgd0moXVbEuI2\npCRl2TBmDQmwOK5jYOREnsXZhMSwCfDtH/dOuJgZz/XMvXM997xf0tG95znPufd7EvGZ4+eec55U\nFZKk5jhh2AVIkpaWwS9JDWPwS1LDGPyS1DAGvyQ1jMEvSQ1j8EtSwxj8ktQwBr8kNcyJwy5gNmvX\nrq3x8fFhlyFJy8auXbv+qqrGeul7XAb/+Pg4k5OTwy5DkpaNJE/32tehHklqGINfkhrG4Jekhjku\nx/hn84tf/IKpqSlefPHFYZcyUKtWrWLdunWsXLly2KVIGlHLJvinpqY4+eSTGR8fJ8mwyxmIquK5\n555jamqKc889d9jlSBpR8w71JDknyf1JnkzyeJKPzdInSf4gyd4kjyW5uGvb9Um+11muX2ihL774\nIqeffvrIhj5AEk4//fSR/1eNpNfavh3Gx+GEE9qv27cP9vt6OeN/Cfg3VfVIkpOBXUl2VtUTXX2u\nBM7rLG8F/hB4a5LTgJuAFlCdfXdU1Y8WUuwoh/6MJhyjpFdt3w5btsDhw+31p59urwNs3jyY75z3\njL+qflBVj3TePw88CZx9RLdrgM9X20PAKUnOAt4N7KyqQ52w3wls6usRSNIyduONr4b+jMOH2+2D\nckxX9SQZBy4C/vyITWcDz3atT3Xa5mqf7bO3JJlMMjk9PX0sZS2ZFStWsHHjRt785jdz4YUX8olP\nfIJXXnnlqPvs37+fO++8c4kqlLTcPPPMsbX3Q8/Bn+SNwJeA36qqnxy5eZZd6ijtr2+smqiqVlW1\nxsZ6uuv4qAYxZvaGN7yB3bt38/jjj7Nz507uvfdebr755qPuY/BLOpr164+tvR96Cv4kK2mH/vaq\n+vIsXaaAc7rW1wEHjtI+UDNjZk8/DVWvjpn18weTM844g4mJCT71qU9RVezfv5+3ve1tXHzxxVx8\n8cU8+OCDANxwww1885vfZOPGjdx2221z9pPUTLfeCqtXv7Zt9ep2+8BU1VEX2mftnwc+eZQ+VwFf\n6fS9FPh2p/004PvAqZ3l+8Bp833nW97yljrSE0888bq2uWzYUNWO/NcuGzb0/BGzWrNmzevaTjnl\nlPrhD39YP/vZz+qFF16oqqo9e/bUzDHcf//9ddVVV/2y/1z9uh3LsUpa/u64o51PSfv1jjuO/TOA\nyZonW2eWXq7quQz4APCdJLs7bf8eWN/5w/EZ4F7gPcBe4DDwm51th5J8HHi4s98tVXVooX+kerWU\nY2bt/73bN5ht3bqV3bt3s2LFCvbs2TNr/177SWqOzZsHdwXPbOYN/qr6FrOP1Xf3KeCjc2zbBmxb\nUHULtH59e3hntvZ+2rdvHytWrOCMM87g5ptv5swzz+TRRx/llVdeYdWqVbPuc9ttt/XUT5IGZSSf\n1bMUY2bT09N8+MMfZuvWrSThxz/+MWeddRYnnHACX/jCF3j55ZcBOPnkk3n++ed/ud9c/SRpqYxk\n8G/eDBMTsGEDJO3XiYnF/1PqhRde+OXlnJdffjlXXHEFN910EwAf+chH+NznPsell17Knj17WLNm\nDQAXXHABJ554IhdeeCG33XbbnP0kaalkZoz6eNJqterIiViefPJJzj///CFVtLSadKyS+iPJrqpq\n9dJ3JM/4JUlzM/glqWGWVfAfj8NS/daEY5Q0XMsm+FetWsVzzz030sFYnefxe4mnpEFaNhOxrFu3\njqmpKY7XB7j1y8wMXJI0KMsm+FeuXOmsVJLUB8tmqEeS1B8GvyQ1jMEvSQ1j8EtSwxj8ktQwBr8k\nNYzBL0kNY/BLUsPMewNXkm3Ae4GDVfVrs2z/t8DMk+5PBM4HxjrTLu4HngdeBl7q9ZGhkqTB6eWM\n/3Zg01wbq+r3q2pjVW0Efgf4P0fMq/vOznZDX5KOA/MGf1U9APQ6Qfp1wF2LqkiSNFB9G+NPspr2\nvwy+1NVcwNeS7EqyZZ79tySZTDI56g9ik6Rh6uePu/8I+L9HDPNcVlUXA1cCH03y63PtXFUTVdWq\nqtbY2Fgfy5Ikdetn8F/LEcM8VXWg83oQuAe4pI/fJ0lagL4Ef5K/Dbwd+G9dbWuSnDzzHrgC+G4/\nvk+StHC9XM55F/AOYG2SKeAmYCVAVX2m0+0fA1+rqp917XomcE+Sme+5s6q+2r/SJUkLMW/wV9V1\nPfS5nfZln91t+4ALF1qYJGkwvHNXkhrG4JekhjH4JalhDH5JahiDX5IaxuCXpIYx+CWpYQx+SWoY\ng1+SGsbgl6SGMfglqWEMfklqGINfkhrG4JekhjH4JalhDH5JahiDX5IaZt7gT7ItycEks86Xm+Qd\nSX6cZHdn+d2ubZuSPJVkb5Ib+lm4JGlhejnjvx3YNE+fb1bVxs5yC0CSFcCngSuBNwHXJXnTYoqV\nJC3evMFfVQ8Ahxbw2ZcAe6tqX1X9HLgbuGYBnyNJ6qN+jfH//SSPJvlKkjd32s4Gnu3qM9Vpm1WS\nLUkmk0xOT0/3qSxJ0pH6EfyPABuq6kLgPwL/tdOeWfrWXB9SVRNV1aqq1tjYWB/KkiTNZtHBX1U/\nqaqfdt7fC6xMspb2Gf45XV3XAQcW+32SpMVZdPAn+TtJ0nl/SecznwMeBs5Lcm6Sk4BrgR2L/T5J\n0uKcOF+HJHcB7wDWJpkCbgJWAlTVZ4B/CvyLJC8BLwDXVlUBLyXZCtwHrAC2VdXjAzkKSVLP0s7o\n40ur1arJyclhlyFJy0aSXVXV6qWvd+5KUsMY/JLUMAa/JDWMwS9JDWPwS1LDGPyS1DAGvyQ1jMEv\nSQ1j8EtSwxj8ktQwBr8kNYzBL0kNY/BLUsMY/JLUMAa/JDWMwS9JDWPwS1LDzBv8SbYlOZjku3Ns\n35zksc7yYJILu7btT/KdJLuTOKWWJB0Hejnjvx3YdJTt3wfeXlUXAB8HJo7Y/s6q2tjrlGCSpMGa\nd7L1qnogyfhRtj/YtfoQsG7xZUmSBqXfY/wfBL7StV7A15LsSrLlaDsm2ZJkMsnk9PR0n8uSJM2Y\n94y/V0neSTv4/0FX82VVdSDJGcDOJH9ZVQ/Mtn9VTdAZJmq1WtWvuiRJr9WXM/4kFwB/DFxTVc/N\ntFfVgc7rQeAe4JJ+fJ8kaeEWHfxJ1gNfBj5QVXu62tckOXnmPXAFMOuVQZKkpTPvUE+Su4B3AGuT\nTAE3ASsBquozwO8CpwP/KQnAS50reM4E7um0nQjcWVVfHcAxSJKOQS9X9Vw3z/YPAR+apX0fcOHr\n95AkDZN37kpSwxj8ktQwBr8kNYzBL0kNY/BLUsMY/JLUMAa/JDWMwS9JDWPwS1LDGPyS1DAGvyQ1\njMEvSQ1j8EtSwxj8ktQwBr8kNYzBL0kNY/BLUsP0FPxJtiU5mGTWOXPT9gdJ9iZ5LMnFXduuT/K9\nznJ9vwqXJC1Mr2f8twObjrL9SuC8zrIF+EOAJKfRnqP3rcAlwE1JTl1osZKkxesp+KvqAeDQUbpc\nA3y+2h4CTklyFvBuYGdVHaqqHwE7OfofEEnSgPVrjP9s4Nmu9alO21ztr5NkS5LJJJPT09N9KkuS\ndKR+BX9maaujtL++sWqiqlpV1RobG+tTWZKkI/Ur+KeAc7rW1wEHjtIuSRqSfgX/DuCfd67uuRT4\ncVX9ALgPuCLJqZ0fda/otEmShuTEXjoluQt4B7A2yRTtK3VWAlTVZ4B7gfcAe4HDwG92th1K8nHg\n4c5H3VJVR/uRWJI0YD0Ff1VdN8/2Aj46x7ZtwLZjL02SNAjeuStJDWPwS1LDGPyS1DAGvyQ1jMEv\nSQ1j8EtSwxj8ktQwBr8kNYzBL0kNY/BLUsMY/JLUMAa/JDWMwS/puLJ9O4yPwwkntF+3bx92RaOn\np6dzStJS2L4dtmyBw4fb608/3V4H2Lx5eHWNGs/4JR03brzx1dCfcfhwu139Y/BLOm4888yxtWth\nDH5Jx43164+tXQvTU/An2ZTkqSR7k9wwy/bbkuzuLHuS/HXXtpe7tu3oZ/GSRsutt8Lq1a9tW726\n3a7+mffH3SQrgE8D7wKmgIeT7KiqJ2b6VNW/7ur/L4GLuj7ihara2L+SJY2qmR9wb7yxPbyzfn07\n9P1ht796uarnEmBvVe0DSHI3cA3wxBz9r6M9GbskHbPNmw36QetlqOds4Nmu9alO2+sk2QCcC3y9\nq3lVkskkDyV531xfkmRLp9/k9PR0D2VJkhail+DPLG01R99rgS9W1ctdbeurqgX8M+CTSf7ebDtW\n1URVtaqqNTY21kNZkqSF6CX4p4BzutbXAQfm6HstcFd3Q1Ud6LzuA77Ba8f/JUlLrJfgfxg4L8m5\nSU6iHe6vuzonya8CpwJ/1tV2apJf6bxfC1zG3L8NSJKWwLw/7lbVS0m2AvcBK4BtVfV4kluAyaqa\n+SNwHXB3VXUPA50PfDbJK7T/yPxe99VAkqSll9fm9PGh1WrV5OTksMuQpGUjya7O76nz8s5dSWoY\ng1+SGsbgl6SGMfglqWEMfklqGINfkhrG4JekhjH4JalhDH5JahiDX5IaxuCXpIYx+CWpYQx+SWoY\ng1+SGsbgl6SGMfglqWEMfklqmJ6CP8mmJE8l2Zvkhlm2/0aS6SS7O8uHurZdn+R7neX6fhYvjbrt\n22F8HE44of26ffuwK9IomHfO3SQrgE8D7wKmgIeT7Jhl7tw/qaqtR+x7GnAT0AIK2NXZ90d9qV4a\nYdu3w5YtcPhwe/3pp9vrAJs3D68uLX+9nPFfAuytqn1V9XPgbuCaHj//3cDOqjrUCfudwKaFlSo1\ny403vhr6Mw4fbrdLi9FL8J8NPNu1PtVpO9I/SfJYki8mOecY9yXJliSTSSanp6d7KEsabc88c2zt\nUq96Cf7M0lZHrP93YLyqLgD+F/C5Y9i33Vg1UVWtqmqNjY31UJY02tavP7Z2qVe9BP8UcE7X+jrg\nQHeHqnquqv6ms/pHwFt63VfS7G69FVavfm3b6tXtdmkxegn+h4Hzkpyb5CTgWmBHd4ckZ3WtXg08\n2Xl/H3BFklOTnApc0WmTNI/Nm2FiAjZsgKT9OjHhD7tavHmv6qmql5JspR3YK4BtVfV4kluAyara\nAfyrJFcDLwGHgN/o7Hsoycdp//EAuKWqDg3gOKSRtHmzQa/+S9WsQ+5D1Wq1anJycthlSNKykWRX\nVbV66eudu5LUMAa/JDWMwS9JDWPwS1LDGPyS1DAGvyQ1jMEvSQ1j8EtSwxj8ktQwBr8kNYzBL0kN\nY/BLUsMY/FpWnHxcWrx5H8ssHS+cfFzqD8/4tWw4+bjUHwa/lg0nH5f6w+DXsuHk41J/9BT8STYl\neSrJ3iQ3zLL9t5M8keSxJP87yYaubS8n2d1Zdhy5r9QrJx+X+mPe4E+yAvg0cCXwJuC6JG86ottf\nAK2qugD4IvAfura9UFUbO8vVfapbDeTk41J/9HJVzyXA3qraB5DkbuAa4ImZDlV1f1f/h4D397NI\naYaTj0uL18tQz9nAs13rU522uXwQ+ErX+qokk0keSvK+uXZKsqXTb3J6erqHsiRJC9HLGX9maatZ\nOybvB1rA27ua11fVgSR/F/h6ku9U1f973QdWTQATAK1Wa9bPlyQtXi9n/FPAOV3r64ADR3ZKcjlw\nI3B1Vf3NTHtVHei87gO+AVy0iHolSYvUS/A/DJyX5NwkJwHXAq+5OifJRcBnaYf+wa72U5P8Suf9\nWuAyun4b0OL4+AJJCzHvUE9VvZRkK3AfsALYVlWPJ7kFmKyqHcDvA28E/ksSgGc6V/CcD3w2ySu0\n/8j8XlUZ/H3g4wskLVSqjr/h9FarVZOTk8Mu47g2Pt4O+yNt2AD79y91NZKGLcmuqmr10tc7d5cp\nH18gaaEM/mXKxxdIWiiDf5ny8QWSFsrgX6Z8fIGkhXIilmXMxxdIWgjP+CWpYQx+SWqYkQl+72KV\npN6MxBi/d7FKUu9G4ozfSbglqXcjEfzexSpJvRuJ4PcuVknq3UgEv3exSlLvRiL4vYtVkno3Elf1\ngHexSlKvRuKMX5LUO4NfkhrG4JekhjH4JalhDH5JapjjcrL1JNPALFOJ92Qt8Fd9LGc58JhHX9OO\nFzzmY7WhqsZ66XhcBv9iJJnsdab5UeExj76mHS94zIPkUI8kNYzBL0kNM4rBPzHsAobAYx59TTte\n8JgHZuTG+CVJRzeKZ/ySpKMYmeBPsinJU0n2Jrlh2PUshSTbkhxM8t1h17IUkpyT5P4kTyZ5PMnH\nhl3ToCVZleTbSR7tHPPNw65pqSRZkeQvkvyPYdeyFJLsT/KdJLuTTA70u0ZhqCfJCmAP8C5gCngY\nuK6qnhhqYQOW5NeBnwKfr6pfG3Y9g5bkLOCsqnokycnALuB9o/z/c5IAa6rqp0lWAt8CPlZVDw25\ntIFL8ttAC/hbVfXeYdczaEn2A62qGvi9C6Nyxn8JsLeq9lXVz4G7gWuGXNPAVdUDwKFh17FUquoH\nVfVI5/3zwJPA2cOtarCq7aed1ZWdZfmfrc0jyTrgKuCPh13LKBqV4D8beLZrfYoRD4SmSzIOXAT8\n+XArGbzOkMdu4CCws6pG/piBTwL/Dnhl2IUsoQK+lmRXki2D/KJRCf7M0jbyZ0VNleSNwJeA36qq\nnwy7nkGrqperaiOwDrgkyUgP6yV5L3CwqnYNu5YldllVXQxcCXy0M5Q7EKMS/FPAOV3r64ADQ6pF\nA9QZ5/4SsL2qvjzsepZSVf018A1g05BLGbTLgKs7Y953A/8wyR3DLWnwqupA5/UgcA/tIeyBGJXg\nfxg4L8m5SU4CrgV2DLkm9Vnnh87/DDxZVZ8Ydj1LIclYklM6798AXA785XCrGqyq+p2qWldV47T/\nW/56Vb1/yGUNVJI1nQsWSLIGuAIY2NV6IxH8VfUSsBW4j/YPfn9aVY8Pt6rBS3IX8GfAryaZSvLB\nYdc0YJcBH6B9Bri7s7xn2EUN2FnA/Ukeo32Cs7OqGnF5Y8OcCXwryaPAt4H/WVVfHdSXjcTlnJKk\n3o3EGb8kqXcGvyQ1jMEvSQ1j8EtSwxj8ktQwBr8kNYzBL0kNY/BLUsP8f5h9s0VOQmX5AAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1acfa027a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# measurements\n",
    "xm = np.array([0,1,2,3,4,5])\n",
    "ym = np.array([0.1,0.2,0.3,0.5,0.8,2.0])\n",
    "\n",
    "plt.plot(xm,ym,'bo',label='Data')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7: Machine Learning\n",
    "\n",
    "Approximate $y = sin(x)$ with an Artificial Neural Network\n",
    "\n",
    "__Trigonometric Function (trig=True)__\n",
    "\n",
    "* Input: $x$\n",
    "* Layer 1: linear layer, 1 node, $l1 = w1 \\; x$\n",
    "* Layer 2: nonlinear layer, 1 node, cosine function, $l2 = \\cos(w2a+w2b\\;l1)$\n",
    "* Layer 3: linear layer, 1 node, $l3 = w3 \\; l2$\n",
    "* Output: $y = \\sum{l3}$\n",
    "\n",
    "__Artificial Neural Network Description (trig=False)__\n",
    "\n",
    "* Input: $x$\n",
    "* Layer 1: linear layer, 2 nodes, $l1 = w1 \\; x$\n",
    "* Layer 2: nonlinear layer, 2 nodes, hyperbolic tangent activation function, $l2 = \\tanh(w2a+w2b\\;l1)$\n",
    "* Layer 3: linear layer, 2 nodes, $l3 = w3 \\; l2$\n",
    "* Output: $y = \\sum{l3}$\n",
    "\n",
    "See [Online Neural Network Demo](https://playground.tensorflow.org) with TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8: Solve Differential Equation(s)\n",
    "\n",
    "Solve the following differential equation with initial condition $y(0) = 5$:\n",
    "\n",
    "$ k \\, \\frac{dy}{dt} = -t \\, y$\n",
    "\n",
    "where $k=10$. The solution of $y(t)$ should be reported from an initial time $0$ to final time $20$. Create of plot of the result for $y(t)$ versus $t$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9: Nonlinear Programming Optimization\n",
    "\n",
    "Solve the following nonlinear optimization problem:\n",
    "\n",
    "$\\min x_1 x_4 \\left(x_1 + x_2 + x_3\\right) + x_3$\n",
    "\n",
    "$\\mathrm{s.t.} \\quad x_1 x_2 x_3 x_4 \\ge 25$\n",
    "\n",
    "$x_1^2 + x_2^2 + x_3^2 + x_4^2 = 40$\n",
    "\n",
    "$1\\le x_1, x_2, x_3, x_4 \\le 5$\n",
    "\n",
    "with initial conditions:\n",
    "\n",
    "$x_0 = (1,5,5,1)$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10: Mixed Integer Nonlinear Programming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11: Optimal Control with Integral Objective\n",
    "\n",
    "__Original Form__\n",
    "\n",
    "$\\min_u \\frac{1}{2} \\int_0^2 x_1^2(t) \\, dt$\n",
    "\n",
    "$\\mathrm{subject \\; to}$\n",
    "\n",
    "$\\frac{dx_1}{dt}=u$\n",
    "\n",
    "$x_1(0) = 1$\n",
    "\n",
    "$-1 \\le u(t) \\le 1$\n",
    "\n",
    "__Equivalent Form for GEKKO with new Variable $x_2$__\n",
    "\n",
    "$\\min_u x_2\\left(t_f\\right)$\n",
    "\n",
    "$\\mathrm{subject \\; to}$\n",
    "\n",
    "$\\frac{dx_1}{dt}=u$\n",
    "\n",
    "$\\frac{dx_2}{dt} = \\frac{1}{2} x_1^2(t)$\n",
    "\n",
    "$x_1(0) = 1$\n",
    "\n",
    "$x_2(0) = 0$\n",
    "\n",
    "$t_f = 2$\n",
    "\n",
    "$-1 \\le u(t) \\le 1$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12: Optimal Control with Economic Objective\n",
    "\n",
    "__Original Form__\n",
    "\n",
    "$\\max_{u(t)} \\int_0^{10} \\left(E-\\frac{c}{x}\\right) u \\, U_{max} \\, dt$\n",
    "\n",
    "$\\mathrm{subject \\; to}$\n",
    "\n",
    "$\\frac{dx}{dt}=r \\, x(t) \\left(1-\\frac{x(t)}{k}\\right)-u \\, U_{max}$\n",
    "\n",
    "$x(0) = 70$\n",
    "\n",
    "$0 \\le u(t) \\le 1$\n",
    "\n",
    "$E=1, \\, c=17.5, \\, r=0.71$\n",
    "\n",
    "$k=80.5, \\, U_{max}=20$\n",
    "\n",
    "__Equivalent Form for GEKKO__\n",
    "\n",
    "$\\min_{u(t)} -J\\left(t_f\\right)$\n",
    "\n",
    "$\\mathrm{subject \\; to}$\n",
    "\n",
    "$\\frac{dx}{dt}=r \\, x(t) \\left(1-\\frac{x(t)}{k}\\right)-u \\, U_{max}$\n",
    "\n",
    "$\\frac{dJ}{dt} = \\left(E-\\frac{c}{x}\\right) u \\, U_{max}$\n",
    "\n",
    "$x(0) = 70$\n",
    "\n",
    "$J(0) = 0$\n",
    "\n",
    "$0 \\le u(t) \\le 1$\n",
    "\n",
    "$t_f = 10, \\, E=1, \\, c=17.5$\n",
    "\n",
    "$r=0.71, \\, k=80.5, \\, U_{max}=20$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13: Optimal Control: Minimize Final Time\n",
    "\n",
    "__Original Form__\n",
    "\n",
    "$\\min_{u(t)} \\; t_f$\n",
    "\n",
    "$\\mathrm{subject \\; to}$\n",
    "\n",
    "$\\frac{dx_1}{dt}=u$\n",
    "\n",
    "$\\frac{dx_2}{dt}=\\cos\\left(x_1(t)\\right)$\n",
    "\n",
    "$\\frac{dx_3}{dt}=\\sin\\left(x_1(t)\\right)$\n",
    "\n",
    "$x(0) = \\left[\\pi/2,4,0\\right]$\n",
    "\n",
    "$x_2\\left(t_f\\right)=0$\n",
    "\n",
    "$x_3\\left(t_f\\right)=0$\n",
    "\n",
    "$-2 \\le u(t) \\le 2$\n",
    "\n",
    "__Equivalent Form for GEKKO__\n",
    "\n",
    "$\\min_{u(t), t_f} \\; t_f$\n",
    "\n",
    "$\\mathrm{subject \\; to}$\n",
    "\n",
    "$\\frac{dx_1}{dt}=t_f \\, u$\n",
    "\n",
    "$\\frac{dx_2}{dt}=t_f \\, \\cos\\left(x_1(t)\\right)$\n",
    "\n",
    "$\\frac{dx_3}{dt}=t_f \\, \\sin\\left(x_1(t)\\right)$\n",
    "\n",
    "$x(0) = \\left[\\pi/2,4,0\\right]$\n",
    "\n",
    "$x_2\\left(t_f\\right)=0$\n",
    "\n",
    "$x_3\\left(t_f\\right)=0$\n",
    "\n",
    "$-2 \\le u(t) \\le 2$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14: PID Control Tuning\n",
    "\n",
    "A [PID Controller](https://en.wikipedia.org/wiki/PID_controller) has proportional, integral, and derivative terms to determine the controller output ($OP$) based on the set point ($SP$) and process variable ($PV$). A standard PID form has constants $K_c$, $\\tau_I$, and $\\tau_D$.\n",
    "\n",
    "\n",
    "$err = SP-PV$\n",
    "\n",
    "$OP = OP_0 + K_c \\, err + \\frac{K_c}{\\tau_I} \\int err \\, dt - K_c \\, \\tau_D \\frac{d\\,PV}{dt}$ \n",
    "\n",
    "The effect of the tuning constants is shown with the [PID Tuning Notebook](http://nbviewer.jupyter.org/url/apmonitor.com/pdc/uploads/Main/pid_widget.ipynb). This example is an alternative implementation in GEKKO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15: Process Simulator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 16: Moving Horizon Estimation\n",
    "\n",
    "Run the Process Simulation cell above to generate the data. The MHE application uses a first order model while the process simulation is a second order system. This is done to emulate a realistic case with model mismatch and measurement noise.\n",
    "\n",
    "This demonstrates just one cycle of an MHE application. Typical MHE applications receive an additional measurements, re-optimize parameters and states, and re-inject the parameters into a controller. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 17: Model Predictive Control"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 18: Debugging Resources\n",
    "\n",
    "Applications may need a more detailed inspection to find errors in programming syntax, errors in modeling assumptions, or to generate good initial guess values. The GEKKO or the solver solution reports syntax errors. Setting m.solve(disp=True) displays the solver output with a message on the line of code that is unsuccessful. Naming the variables such as name='state' is helpful to display the equations in a readable form.\n",
    "\n",
    "Other strategies for obtaining a successful solution include:\n",
    "\n",
    "* Increase the number of iterations with MAX_ITER (0-1000+)\n",
    "* Increase the diagnostic level with DIAGLEVEL (0-10)\n",
    "* Change the solver with SOLVER (1-5)\n",
    "* Calculate model SENSITIVITY (1)\n",
    "* Solve a square problem with # Variables = # Equations\n",
    "* Set COLDSTART to initialize problem (0-2)\n",
    "\n",
    "Additional [modeling](http://apmonitor.com/do/index.php/Main/ModelFormulation), [initialization](http://apmonitor.com/do/index.php/Main/ModelInitialization), and [decomposition](https://www.sciencedirect.com/science/article/pii/S0098135415001179) tips may be helpful. There is also an online [discussion group](http://apmonitor.com/wiki/index.php/Main/UsersGroup), [video playlist](https://www.youtube.com/playlist?list=PLLBUgWXdTBDjcqDl2e5F_hcBjEc6vjr1X), [GEKKO documentation](http://gekko.readthedocs.io/en/latest/), and [APMonitor documentation](http://apmonitor.com/wiki/index.php/Main/HomePage) as additional resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
